1. Field of the Invention
The present invention generally relates to a method and apparatus for remote control of one or more computer systems by means of replications of actions.
2. Related Art
Consider the problem of remotely controlling a computer system (denoted by the term controlled computer system) by interacting with its graphical user interface (GUI). The term remotely refers to the situation in which the user interacts directly with a different computer system (denoted by the term controlling computer system). This is a common problem, for which a variety of solutions exist.
A first class of approaches well known in the art consists of rendering individual components of the GUI of the controlled computer system on the controlling computer system. An example of this class of solutions is the X-windows windowing system, where the controlling computer executes an X-server, which renders GUI components of the controlled systems. In this case, the GUIs of applications running on the controlled system are displayed on the screen of the controlling system by the X-server. Using these approaches, the user actually interacts with the GUI of applications running on the controlled system. One main limitation of these approaches is that the operating system of the controlled system must support this type of GUI rendering. Common operating systems, like several members of the Microsoft Windows (Microsoft, Windows and related terms are trademarks of Microsoft, Inc. of Redmond, Wash.) family of operating systems, do not support natively this approach.
A second class of approaches consists of replicating on the controlling computer system the content of the GUI of the controlled computer system, and providing mechanisms for the user to interact with this replica of the GUI. This approach is used, for example, by the freeware software package called VNC and by common e-meeting software that allow users to remotely interact with the GUI of the computer of other users. In these approaches, the GUI of the controlled system is rendered on the controlled system itself (unlike in the case of X-Windows, where it is actually rendered on the controlling system). A program on the controlled system produces a description of the visual content of the GUI (for example, a bitmap), and sends it to the controlling system. On the controlling system, another program renders this description to the user and observes the user actions on the rendered description. When the user hits a key or performs a mouse operation, the program on the controlling system sends to the program on the controlled system a description of the action: for example, when the user clicks the mouse, the description contains the button (right-center-left), the click (up-down) and the position in pixel of the mouse. The program on the controlled system identifies the widget the user wants to interact with, for example using the coordinates of the mouse, and performs the desired action. The program on the controlled system sends updated descriptions of the GUI content to the program on the controlling system. In these approaches, the user does not interact directly with the GUI of the controlled system, but the programs on the controlling and controlled system provide the user with the illusion of a direct interaction. One main limitation of these approaches is the need of a fast and reliable network link between the controlled and controlling systems (in order to ensure an acceptable user experience).
Clearly, neither approach is well-suited for simultaneous control of multiple controlled systems. In particular, neither approach supports the ability of simultaneously performing the same action on multiple controlled computer systems where the action consists of GUI interactions.
There are two main approaches to the problem of replicating the same action across multiple controlled computer systems. The first approach consists of writing a program that performs the desired interaction, distributing the program to each controlled computer system, and executing it. The main limitation of this approach is the cost: writing and testing a program is an expensive and time-consuming operation, and is therefore suitable for complex or repetitive operations that can be planned long before they are executed.
The second approach consists of using a macro recorder. A macro recorder observes the interactions of the user with the GUI in terms of low-level operations, and records them in a format that is then executable by an appropriate program, potentially on a different system. Macro recorders typically use very low-level description of actions (e.g., “perform a mouse click at a certain coordinate relative to the last created window”). The main limitation of macro recorders is brittleness: changes in the environment (for instance, in the position, size, or appearance of windows) confuse a macro recorder, and the execution consequently fails to complete correctly. Both macro recorders and programs are also brittle with respect to unexpected circumstances. This brittleness can be reduced in programs by increasing the complexity of the design to cope with different failure types.
In view of the foregoing there exists a need for a mechanism for remotely controlling a GUI by local interactions.